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Method and System for Generating 
Annotations for Video 



NOTICE OF COPENDING APPLICATION 
5 U.S. patent application entitled "METHOD AND APPARATUS FOR 

AUTOMATICALLY ABSTRACTING MULTIMEDIA INFORMATION," Application 

No. , filed on September 10, 1999, assigned to the assignee of the application herein 

is referred to below. 



10 FIELD OF THE INVENTION 

The present invention relates to the field of video indexing, retrieval and 
abstraction. More particularly, the present invention relates to a system and method for 
on-the-fly video annotations for enabling improved video indexing and retrieval. 

15 BACKGROUND OF THE INVENTION 

For many years, an increasing number of people own and use video 
recorders to make video movies that capture their experiences and document their lives. 
Oddly, most videos are put into a storage box and rarely watched again. 

Research is growing in the field of video abstracting. Video abstracting is 

20 the processes of taking unedited video footage and combining shorter segments of that 
footage into one abstract. Existing automatic video abstracting systems concentrate on 
feature films, documentaries or newscasts. Currently, there are generally three systems 
that produce videos as abstracts. The first is called video skimming. It aims mainly at 
abstracting documentaries and newscasts. Video skimnaing assumes that the audio track 

25 transcript is available. The video and the transcript are then aligned by word spotting. 
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The audio track of the video skim is constructed by using language analysis (such as the 
Term Frequency Inverse Document Frequency measure) to identify important words in 
the transcript. Audio clips around those words are then cut out. Based on detected faces, 
text, and camera operations, video clips for the video skim are selected from the 
surrounding frames. 

The second system called MoCA Abstracting. MoCA Abstracting was 
explicitly designed to generate trailers of feature films. The MoCA Abstracting system 
performs an extensive video analysis of a feature film to segment it into shots or scenes 
and to determine special events, such as text appearing in the title sequence, close-up 
shots of main actors, explosions, gunfire, etc. This information is used to select the clips 
for the video abstract. During the final assembly, ordering and editing rules are 
presented. Since MoCA Abstracting relies highly on special events such as explosions, 
gunfire, shot or reverse shot dialogs, and actions that are usually not present in home 
videos it cannot be used to abstract home video. 

The third system by Saarela and Merialdo does not perform any automatic 
content analysis. Instead they assume that videos have been annotated manually or 
automatically by descriptors for various properties and relations of audio and video 
segments. Based on those descriptors the authors try to define "optimal" summaries. 
They present constraints for video summaries and methods to evaluate the importance of 
a specific segment. 

These existing automatic video abstracting systems concentrate on feature 
films, documentaries or newscasts. Since raw video footage such as home video is 
inherently different from all broadcast video, new abstracting principles and algorithms 
are needed. 
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SUMMARY OF THE INVENTION 

The present invention provides a method and system for processing video 
signals. The system receives a video signal, a first audio signal containing an annotation 
and a second audio signal containing environmental sounds corresponding to the video 
signal. In one embodiment the system generates searchable annotations corresponding to 
the video and second audio signals via the first audio signal. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



Figure 1 illustrates a block diagram of an embodiment of a system 
utilizing the present invention. 

5 

Figure 2 illustrates a flow diagram of the steps performed in one 
embodiment of the present invention. 

Figure 3 illustrates a block diagram of adaptive noise cancellation for 
10 removing the annotations from the audio signal, according to one embodiment of the 
present invention. 

Figure 4 illustrates the method for localizing annotations by signal energy, 
according to one embodiment of the present invention. 

15 

Figure 5 illustrates a three-step process for generating enhanced video 
abstracts, according to one embodiment of the present invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 



Embodiments of the present invention relate to a method and system for 
generating annotations for video indexing and retrieval. More particularly, in one 
5 embodiment the present invention relates to a system and method for on-the-fly video 
annotations for enabling improved and more selective video abstracts. On-the-fly 
annotations are verbal notes made by the user contemporaneously with the video footage. 
Although described with respect to video abstracting, the present invention can be 
implemented in applications that benefit from indexing and retrieval via annotations. In 

10 the following description, for purposes of explanation, specific nomenclature is set forth 
to provide a thorough understanding of the present invention. For example, the term 
"video" comprises motion pictures and sound. A "video abstract" denotes a short video 
consisting of motion pictures and sound. However, it will be apparent to one skilled in 
the art that these specific details are not required in order to practice the present 

15 invention. 



The present system allows a user to insert annotations within video 
footage, use the annotations to create video abstracts and remove the annotations from the 
original video. The invention may be implemented in multiple embodiments. For 

20 example, the video may be annotated "on-the-fly" as described below; alternatively the 
video may be annotated at a later time. Furthermore, the annotations are not limited to 
video, but may also be extended to purely audio applications and video obtained by other 
means. Although the embodiments below implement both hardware components and 
software components, it will be apparent one skilled in the art that these specific details 

25 are not required in order to practice the present invention. 
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Video Processing System 

Referring to FIG. 1, an illustrative embodiment of a system for generating 
annotations for video indexing and retrieval is shown. The following discussion of the 
5 annotation system is described, as it would be implemented when generating a video 

abstract, by way of example only. One of ordinary skill in the art could implement further 
embodiments of the annotation system for other purposes. 

In one embodiment, the following system utilizes a two-microphone setup 
with a video recorder to support annotations on-the-fly. The electronic system (100) 
10 comprises a processor unit (110), a display device (120), a data storage unit (130), an 

analog to digital converter (A/D) (140), a video recorder (150), a voice microphone (160), 
and an environmental microphone (170). 

The processor unit (1 10) acquires a digital video signal (141) from the 
video recorder (150) via the A/D (140). Voice microphone (160) provides digitized audio 
15 annotations via the A/D (140). Environmental microphone (170) provides digitized 
environmental audio via A/D (140) containing both environmental sounds as well as 
annotations. The processor unit (110) takes the digital video signal, analyzes it and stores 
the video together with the digitized voice and environmental audio signals in the data 
storage (130). When a user requests a new video abstract, the digitized voice signal, 
20 environmental audio signal and digitized video signal (141) are retrieved from the data 
storage (130) in order to create a new video abstract. The video abstract is played back 
on the display device (120). 

The invention would be equally applicable to a system that utilizes a 
digital video camera with two digital audio tracks or other digital or analog recording 
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device. Further, the invention could be implemented in a system using less than or more 
than two microphones. 

Still referring to FIG, 1, a video recorder with two microphones is setup to 
support annotations on-the-fly. The voice microphone (160) may be a unidirectional head 

5 worn speech input microphone that captures the voice of the cameraman. The output of 
the voice microphone (160) is stored as the left audio channel of the video recorder (150), 
while the output of the environmental microphone is stored as the right audio channel of 
the video recorder (150). Once the video has been transferred to the processor unit (110), 
the voice channel is transcribed to computer readable text using a voice to text conversion 

10 system. In another embodiment, the audio channels may be switched. The present 
system can be implemented with existing video recorders, without modification or 
redesign. 



The Processor Unit 

15 FIG. 2 illustrates a flow diagram of the operations performed by the 

processor unit (1 10), according to one embodiment of the present invention. In 
alternative embodiments, the operations shown in FIG. 2 could be performed by other 
types of circuitry, such as a DSP, ASIC, or other dedicated circuitry. Moreover, the 
operations could be a set of instructions stored on a machine readable medium to be 

20 executed by the processor unit (110). 

At processing block 210 the digital signal (141) is received by a splitter 
that separates the audio and video channels. At processing block 220 the processor unit 
(110) removes on-the-fly annotations from the digitized environmental audio. Shot 
boundaries are generated through video processing block 250 and are used to shorten the 

25 duration of each shot. Then, the annotation-free audio signal and shot boundaries are 
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used to find selected clips of video footage. This process is carried out at shot shortening 
block 230. Next, the description of a shot's representative clips is stored in the data 
storage (130). In processing block 240, the digitized annotations (161) are passed on to a 
speech recognition engine. The resulting transcribed annotations as well as time-stamp 
5 annotations are stored in data storage (130). 



Annotation Transcription 

Annotations may be varying types of free speech. However, the system 
allows each content descriptive free speech annotation to be preceded by a keyword 

10 specifying the kind of annotation and its associated temporal validity. In one 

embodiment, for the processor unit (1 10) to transcribe annotations with the speech 
recognition engine (240), the syntax of the annotations is the following: <keyword>, 
<temporal validity>, and <free speech text> where keyword is a title, person or object and 
temporal validity is either of a shot, action, event, day or week. The keyword allows 

15 general descriptions to be distinguished from specific descriptions (e.g. naming persons 
and objects). In practice, object descriptions are most often defaulted if no keyword is 
present. The temporal validity specifies the temporal extent of the description. By way 
of example, to label a recording as a one- week vacation video, you would say "TITLE 
WEEK..." in order to assign the annotation to the vacation video. 

20 In one embodiment, the names SHOT, ACTION, EVENT, DAY and 

WEEK denote five types of time-based hierarchical shot clusters. An ACTION or level 1 
cluster is defined as a sequence of contiguous actions. Shots within a 5-minute duration, 
form an ACTION cluster. The idea behind this type of cluster is to group shots of high 
causality. An EVENT or level 2 cluster represents a sequence of contiguous activities 

25 within an one-hour duration. EVENT activities usually last longer than actions and span 
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numerous actions. A DAY or level 3 cluster represents individual days, while a WEEK 
or level 4 cluster represents seven contiguous DAYs. In one embodiment, if no temporal 
validity is provided, SHOT or level 0 is assumed. As a result, annotations are not 
assigned accidentally to unrelated video clips. 

5 Editing commands are also supported through Annotations. The first 

supported command is DELETE. For the processor unit (1 10) to DELETE sections of 
video, the syntax of the annotations is the following: DELETE, <time> and <temporal 
validity> where time is either CURRENT or LAST. For example, a delete command 
could be "DELETE CURRENT SHOT," or "DELETE LAST EVENT." Another 

10 supported editing command is for RATING. For the processor unit (1 10) to rate the 
importance of contents, the syntax of the annotations may be the following: RATING, 
<timextemporal validity> as <rating> where rating is either SECONDARY, DEFAULT, 
IMPORTANT or HIGHLIGHT. In other embodiments, alternative editing commands can 
be supported without departing from the present invention. 

15 

Enhancing Video Abstracts 

Referring to FIG*2, which illustrates a flow diagram of the processor unit's 
(1 10) functions, annotations enable semantic indexing and retrieval and are useful in 
diverse applications. Basically, annotations can be exploited in many ways to enhance 

20 the video extracts in the video processing block 250. For example, an annotation can be 
used to select the source video material that is to be abstracted allowing a viewer to watch 
thematic abstracts. Hereto, the user specifies a text query. All shot clusters whose 
annotations meet the query are taken as the source video material. For instance, assume 
that a user wants to view an abstract of all birthday parties of his daughter Nadine, By 

25 specifying "Nadine AND birthday" as the query, all clusters containing both words in 
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their associated annotation(s) will be selected as the source video set. An abstract is then 
created based on the source video set. The user may also specify that the cluster level of 
clusters retrieved by some query words should be increased for certain query terms with 
respect to the actual cluster level to which the query term is assigned. For example, if 
only one shot is annotated "Nadine's birthday," but more than one shot has been recorded, 
you can request to retrieve the level 2 cluster associated with "Nadine's birthday" which 
may generate all the shots of the birthday. 

Secondly, annotations can be used to enhance the visual presentation of 
video abstracts. Hereto, short TITLE annotations may be shown as centered text title 
introducing the subsequent video clips. Annotations might also be added as small 
scrolling text banners at the bottom or top of the video frames. PEOPLE annotations can 
be used to append a closing sequence to the video abstract. 

Finally, shots can be cut down to the part(s) with descriptive annotations, 
and annotation-free shots may be discarded. This abstraction rule is motivated by the fact 
that the important objects, persons, or actions often come along with annotations. 
Nevertheless, additional clips may either be discarded or added in order to meet the target 
duration of the requested abstract. 



Removing Annotations 

Referring now to FIG. 3 which illustrates a block diagram of adaptive 
noise cancellation for removing the annotations from the audio signal in two steps 
according to one embodiment. In the first step, an annotation detector examines the audio 
signal in a window of a one second duration and counts the number of amplitude values 
above a threshold. In alternate embodiments the annotation detector may examine the 
audio signal in windows less than or greater than a one second duration. If the count 
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exceeds a certain number, the window is declared to contain annotations. Then each 
annotation window is expanded by half a second to the left and a second to the right in 
order not to cut off the beginning and ending of the annotation. Overlapping annotation 
windows are merged into one large annotation's occurrence. The processor unit (110) 
5 removes annotations from the environmental audio signal. The least-mean-square (LMS) 
algorithm developed by Widrow and Hoff may be used to remove the annotations. In 
other embodiments, annotations may be removed by other methods or algorithms. In 
LMS filtering a transversal filter (310) estimates the noise signal (e.g. the signal of 
annotations) in the environmental sound. The difference between the environmental 

10 sound and the estimate of annotations is used to calculate the estimation error. Based on 
the estimation error, the adaptive filter (320) then adjusts the tap weights to reduce the 
mean-square value of the estimation error. 

One potential problem of the LMS algorithm is its slow adjustment rate to 
the real tap weights of a wide-sense stationary signal as well as to system changes of the 

15 underlying signal. Referring to FIG. 4, in order to improve the quality of removal at the 
beginning of each annotation segment, noise cancellation is not only performed forward 
in time but also backward over all detected annotation ranges (410). The backward noise 
cancellation is used to estimate the annotation-free audio signal for the first half of each 
annotation segment (430). While the forward noise cancellation is used to estimate the 

20 annotation-free audio signal for the second half of each annotation segment (440). This 
scheme improves the beginning of annotation segments. 

Creating Video Abstracts 

Varying methods for creating video abstracts may be implemented with 
25 the present system. By way of example, a video abstract may be obtained from the 
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methods disclosed in co-pending provisional U.S. patent application entitled "METHOD 
AND APPARATUS FOR AUTOMATICALLY ABSTRACTING MULTIMEDIA 

INFORMATION." Application No. , filed on September 10, 1999, assigned to the 

assignee of the application herein. 
5 In one embodiment, video abstracts are created in 3 steps. First a one-time 

processing step analyzes the video in order to determine Shot Boundaries (511), Time and 
Date of Recording (512), Hierarchical Shot Clusters (513), and Short Representative 
Clips (514). 

Shot Boundaries (51 1) are determined based upon how the video was 

10 acquired, whether with a digital video camera or analog video recorder. If a digital video 
camera is used, the Shot Boundaries (511) are calculated based upon the time and date of 
the recording stored with the frames. In one embodiment, time increments of more than 
one second between contiguous frames marks a Shot Boundary (511). With analog video 
cameras, video is first digitized and Shot Boundaries (51 1) are then determined. 

15 Ti me and Date of Recording (5 1 2) are also determined based upon how 

the video was acquired, whether with a digital video camera or analog video recorder. If 
a digital video camera is used, the Time and Date of Recording (512) can be read on the 
digital video stream. In the case of an analog video recorder, a superimposed date and 
time stamp created on the film when the video was acquired can be extracted. By way of 

20 example, this information may be extracted from the video by adjusting the text 

segmentation and text recognition algorithms to the specific characteristics of time and 
date information in the videos. 

Hierarchical Shot Clusters (513) are constructed based upon the Time and 
Date of Recording (512), thus generating a four-level Hierarchy of Shot Clusters (513). 

25 The Shot Clusters (5 1 3) are determined by temporal distance between contiguous shots. 
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Individual shots represent a level 0 cluster, while levels 1, 2, 3, and 4 represent a 
sequence of contiguous actions, activities, individual days and weeks. 

In one embodiment, Short Representative Clips (514) for shots are created 
when a shot is reduced to a relevant part based on audio features. By way of example, the 
5 shot may be reduced to approximately 10 seconds of video. Finally, these representative 
clips that are to be part of the video abstract are selected and compiled into the final video 
abstract by inserting edits and title sequences (530). Clip selection and edit decisions are 
based upon heuristic rules. For example, shots of different days are always spliced 
together by a fade out or fade in sequence making the temporal gap explicitly visible. 

10 The foregoing has described a method and system for creating video 

abstracts. More particularly, the present invention relates to a system and method for on- 
the-fly video annotations for enabling improved and more selective video abstracts. It is 
contemplated that changes and modifications may be made by one of ordinary skill in the 
art, to the materials and arrangements of elements of the present invention without 

15 departing from the scope of the invention. 
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CLAIMS 

We claim: 

1 LA method for processing video comprising: 

2 receiving a video signal; 

3 receiving a first audio signal containing annotations; 

4 receiving a second audio signal containing environmental sounds corresponding to 

5 the video signal; and 

6 generating searchable annotations corresponding to the video and second audio 

7 signals via the first audio signal 

1 2. The method as claimed in claim 1 further comprising: removing 

2 the annotations from the second audio signal. - — ^ 

1 3. The method as claimed in claim 2 wherein removing the 

2 annotation from the second audio signal further comprises: utilizing a least-mean- 

3 square algorithm. 

1 4. The method as claimed in claim 1 further comprising: 

2 generating a center text title via the searchable annotations; and 

3 generating a scrolling text banner via the searchable annotations. 

1 5. The method as claimed in claim 1 further comprising: 
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2 generating a video abstract via the first and second audio signals, the video signal 

3 and the searchable annotations. 

1 6. A system for processing video comprising: 

2 means for receiving a video signal; 

3 means for receiving a first audio signal containing annotations; - 

4 means for receiving a second audio signal containing environmental sounds 

5 corresponding to the video signal; and 

6 means for generating searchable annotations corresponding to the video and 

7 second audio signals via the first audio signal. 

1 7. The system as claimed in claim 6 further comprising: 

2 removing the annotations from the second audio signal. 

1 8. The system as claimed in claim 6 further comprising: 

2 means for generating a center text title with thecomputer searchable annotations; 

3 and 

4 means for generating a scrolling text banner with the computer searchable 

5 annotations. 

1 9. The system as claimed in claim 6 further comprising: 

2 means for generating a video abstract via the first and second audio signals, the 

3 video signal and the searchable annotations. 
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1 10. A system for processing video comprising: 

2 a video signal; ^ ^ 

3 a first audio signal containing annotations; 

4 a second audio signal containing environmental sounds corresponding to the video 

5 signal; and 

6 searchable annotations corresponding to the video and second audio signals 

7 generated via the first audio signal. 

1 11. The system as claimed in claim 10 wherein the processor removes 

2 the annotations from the second audio signal. 

1 12. The system as claimed in claim 10 wherein the processor: 

2 generates a center text title with the computer searchable annotations; and 

3 generates a scrolling text banner with the computer searchable annotations. 

1 13. The system as claimed in claim 1 0 wherein the processor: 

2 generates a video abstract via the first and second-audio signals, the video signal 

3 and the searchable annotations. 

1 14. The system as claimed in claim 10 wherein the video signal is 

2 received from a video recorder. 
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1 15. The system as claimed in claim 10 wherein the first and second 

2 audio signals are received from at least one microphone. 



1 16. A computer-readable medium having stored thereon a plurality of 

2 instructions, said plurality of instructions when executed by a computer, cause 

3 said computer to perform the method of: 

4 receiving a video signal; ^ 

5 receiving a first audio signal containing annotations; 

6 receiving a second audio signal containing environmental sounds corresponding to 

7 the video signal; and 

8 generating searchable annotations corresponding to the video and second audio 

9 signals via the first audio signal. 



17. The computer-readable medium of claim 16 having stored thereon 
additional instructions, said additional instructions, said plurality of instructions 
when executed by a computer, cause said computer to further perform the method 



4 of removing the annotations from the second audio signal. 

1 18. The computer-readable medium of claim 16 having stored thereon 

2 additional instructions, said additional instructions, said plurality of instructions 

3 when executed by a computer, cause said computer to furtherperform the method 

4 of: 

5 generating a center text title via the searchable annotations; and 



generating a scrolling text banner via the searchable annotations. 
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1 19. The computer-readable medium of claim 16 having stored thereon- 

2 additional instructions, said additional instructions when executed by a computer, 

3 cause said computer to further perform the method of; 

4 generating a video abstract via the first and second audio signals, the video signal 

5 and the searchable annotations. 
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ABSTRACT OF THE DISCLOSURE 



A method and system for processing video signals is disclosed. The 
system receives a video signal, a first audio signal containing an annotation and a second 
audio signal containing environmental sounds corresponding to the video signal. In one 
embodiment the system generates searchable annotations corresponding to the video and 
second audio signals via the first audio signal. 
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DECLARATION AND POWER OF ATTORNEY FOR PATENT APPLICATION 
(FOR INTEL CORPORATION PATENT APPLICATIONS) 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below, next to my name. 

I believe I am the original, first, and sole inventor (if only one name is listed below) or an original, 
first, and joint inventor (if plural names are listed below) of the subject matter which is claimed and 
for which a patent is sought on the invention entitled 

Method And System For Generating Annotations For Video 



the specification of which 

X is attached hereto. 

was filed on as 

United States Application Number 

or PCT International Application Number 

and was amended on 

(if applicable) 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claim(s), as amended by any amendment referred to above. I do not 
know and do not believe that the claimed invention was ever known or used in the United States of 
America before my invention thereof, or patented or described in any printed publication in any 
country before my invention thereof or more than one year prior to this application, that the same 
was not in public use or on sale in the United States of America more than one year prior to this 
application, and that the invention has not been patented or made the subject of an inventor's 
certificate issued before the date of this application in any country foreign to the United States of 
America on an application filed by me or my legal representatives or assigns more than twelve 
months (for a utility patent application) or six months (for a design patent application) prior to this 
application. 

I acknowledge the duty to disclose all information known to me to be material to patentability as 
defined in Title 37, Code of Federal Regulations, Section 1 .56. 

I hereby claim foreign priority benefits under Title 35, United States Code, Section 1 19(a)-(d), of any 
foreign appiication(s) for patent or inventor's certificate listed below and have also identified below 
any foreign application for patent or inventor's certificate having a filing date before that of the 
application on which priority is claimed: 
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Prior Foreign Application(s) 



Priority 
Claimed 



(Number) (Country) (Day/Month/Year Filed) Yes No 



(Number) (Country) (Day/Month/Year Filed) Yes No 



(Number) (Country) (Day/Month/Year Filed) Yes No 



I hereby claim the benefit under Title 35, United States Code, Section 1 19(e) of any United States 
provisional application(s) listed below: 



Application Number Filing Date 



Application Number Filing Date 



I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
application(s) listed below and, insofar as the subject matter of each of the claims of this application 
is not disclosed in the prior United States application in the manner provided by the first paragraph 
of Title 35, United States Code, Section 1 12, 1 acknowledge the duty to disclose all information 
known to me to be material to patentability as defined in Title 37, Code of Federal Regulations, 
Section 1.56 which became available between the filing date of the prior application and the national 
or PCT international filing date of this application: 



Application Number Filing Date Status - patented, 

pending, abandoned 



Application Number 



Filing Date 



Status -- patented, 

pending, abandoned 



I hereby appoint the persons listed on Appendix A hereto (which is incorporated by reference and a 
part of this document) as my respective patent attorneys and patent agents, with full power of 
substitution and revocation, to prosecute this application and to transact all business in the Patent 
and Trademark Office connected herewith. 

Send correspondence to John P. Ward , BLAKELY, SOKOLOFF, TAYLOR & 

(Name of Attorney or Agent) 
ZAFMAN LLP, 12400 Wilshire Boulevard 7th Floor, Los Angeles, California 90025 and direct 

telephone calls to John P. Ward , (408) 720-8598. 

(Name of Attorney or Agent) 

I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 

Full Name of Sole/First Inventor Rainer Wolfgang Lienhart 



Inventor's Signature Date . 



Residence Santa Clara. California Citizenship Germany 



(City, State) (Country) 

Post Office Address 2727 Forbes Avenue 

Santa Clara, California 95051 

Full Name of Second/Joint Inventor Boon-Lock Yeo 



Inventor's Signature Date . 



Residence Sunnyvale. California Citizenship Singapore 

(City, State) (Country) 

Post Office Address 969 Sutter Avenue 

Sunnyvale. California 94086 
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William E Alford, Reg. No. 37,764; Farzad E. Amini, Reg. No. P42.261; Aloysius T. C. AuYeung, Reg. No. 
35 432- William Thomas Babbitt, Reg. No. 39,591 ; Carol F. Barry, Reg. No. 41 ,600; Jordan Michael 
Becker Reg No. 39,602; Bradley J. Bereznak, Reg. No. 33,474; Michael A. Bernadicou, Reg. No. 35,934; 
Roger W Blakely, Jr., Reg. No. 25,831; Gregory D. Caldwell, Reg. No. 39,926; Ronald C. Card, Reg. No. 
P44 587- Thomas M. Coester, Reg. No. 39,637; Stephen M. De Klerk, under 37 C.F.R. § 10.9(b); Michael 
AnthonyDeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew Diehl, Reg. No. 
40 992- Matthew C. Fagan, Reg. No. 37,542; Tarek N. Fahmi, Reg. No. 41 ,402; James Y. Go, Reg. No. 
40 621 '• James A. Henry, Reg. No. 41 ,064; Willmore F. Holbrow III, Reg. No. P41 ,845; Sheryl Sue 
Ho'lloway, Reg. No. 37,850; George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 30,139; Dag 
H Johansen, Reg. No. 36,172; William W. Kidd, Reg. No. 31,772; Erica W. Kuo, Reg. No. 42,775; Michael 
J Mallie Reg. No. 36,591; Andre L. Marais, under 37 C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 
42 879- Darren J. Milliken, Reg. 42,004; Lisa A. Norris, Reg. No. P44.976; Chun M. Ng, Reg. No. 36,878; 
Thien T. Nguyen, Reg. No. 43,835; Thinh V. Nguyen, Reg. No. 42,034; Dennis A. Nicholls, Reg. No. 
42 036- Kimberley G. Nobles, Reg. No. 38,255; Daniel E. Ovanezian, Reg. No. 41,236; Babak Redjaian, 
Reg No 42,096; William F. Ryann, Reg. 44,313; James H. Salter, Reg. No. 35,668; William W. Schaal, 
Reg No. 39,018; James C. Scheller, Reg. No. 31,195; Jeffrey Sam Smith, Reg. No. 39,377; Maria 
McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, Reg. No. 25,128; Judith A. Szepesi, Reg. No. 
39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. Taylor, Reg. No. 25,129; John F. Travis, Reg. 
No 43,203; George G. C. Tseng, Reg. No. 41 ,355; Joseph A. Twarowski, Reg. No. 42,1 91 ; Lester J. 
Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 41,364; John Patrick Ward, Reg. No. 40,216; 
Charles T. J. Weigell, Reg. No. 43,398; Kirk D. Williams, Reg. No. 42,229; James M. Wu, Reg. No. 
P45.241; Steven D. Yates, Reg. No. 42,242; Ben J. Yorks, Reg. No. 33,609; and Norman Zafman, Reg. 
No. 26,250; my patent attorneys, and Andrew C. Chen, Reg. No. 43,544; Justin M. Dillon, Reg. No. 
42 486; Paramita Ghosh, Reg. No. 42,806; and Sang Hui Kim, Reg. No. 40,450; my patent agents, of 
BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th 
Floor, Los Angeles, California 90025, telephone (310) 207-3800, and Alan K. Aldous, Reg. No. 31 ,905; 
Robert D. Anderson, Reg. No. 33,826; Joseph R. Bond, Reg. No. 36,458; Richard C. Calderwood, Reg. 
No. 35,468; Jeffrey S. Draeger, Reg. No. 41 ,000; Cynthia Thomas Faatz, Reg No. 39,973; Sean 
Fitzgerald, Reg. No. 32,027; Seth Z. Kalson, Reg. No. 40,670; David J. Kaplan, Reg. No. 41 ,105; Charles 
A Mirho, Reg. No. 41,199; Leo V. Novakoski, Reg. No. 37,198; Naomi Obinata, Reg. No. 39,320; 
Thomas C. Reynolds, Reg. No. 32,488; Kenneth M. Seddon, Reg. No. 43,105; Mark Seeley, Reg. No. 
32,299; Steven P. Skabrat, Reg. No. 36,279; Howard A. Skaist, Reg. No. 36,008; Steven C. Stewart, Reg. 
No. 33,555; Raymond J. Werner, Reg. No. 34,752; Robert G. Winkle, Reg. No. 37,474; and Charles K. 
Young, Reg. No. 39,435; my patent attorneys, and Thomas Raleigh Lane, Reg. No. 42,781 ; Calvin E. 
Wells; Reg. No. P43.256, Peter Lam, Reg. No. P44.855; and Gene I. Su, Reg. No. 45,140; my patent 
agents, of INTEL CORPORATION; and James R. Thein, Reg. No. 31 ,710, my patent attorney; with full 
power of substitution and revocation, to prosecute this application and to transact all business in the 
Patent and Trademark Office connected herewith. 
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Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be material to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be material to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1 .98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1 ) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1 ) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1 ) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 



